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ABSTRACT 

, ^ , After an extensive review of the literature concerned 

with evaluating career objectives, a literature with little cohesion, 
SIX areas were isolated as crucial indices of the adequacy of career 
education. These areas are; (1) self-awareness, (2) career-awareness, 
(3) decision-making skills, <a) overall school behavior, (5) career 
competencies, and (6) career attitudes and interests. Measures in all 
six areas should be obtained for the evaluation of career education 
programs. Preferably, measures should be obtained through 




statistically analyzed. Special emphasis is given to the equality of 
the control group with the experimental group because of validity 



problems related to the maturation bias. The importance of research 
follow-up is also stressed. Guidelines and recommendations are 
offered to reduce the amount of error in the evaluation of career 
education programs. (BJG) 



^ iiP.ST COPY mml^ 

rv 
o 



o 



EDUCATION 4 \^Cl.FAtt(> 
HATtONAl iNSTlTUTii OF 

♦h- :>^;. SMS' '«A,> lUI S KM 

'Ml »M .»w . 'V. ,AN /.\ '-.W . .-N 

i I' ;u> ' f * - AW . > 1. 1 t'u'i 
-IS' it ... S"' '*SA. S'.' • M • 



Evaluating Career Education: A Review and Model' 

by 

Bruce U. Tuckman and Jof.eph A. Carducci 



rH 

a 
© 



ERIC/TM Report 39 
December 1974 



6-^ 

T«cfc ^ll!/®''®^^ upon Which this article Is based was funded by the ERIC Clearinghouse on 
IrH^ia^?'"^??'?^ and Evaluation. The Clearinghouse operates under contract with thi 
National Institute of Education, U.S. Department of Health. Education and Welfare. 

nSn^Icc ?i;Lr^![^?''^'!^.'"^\Pr°^®^^' ""^^'^ government sponsorship are encouraged to 

^;®ely their judgment In professional and technical matters. Points of view or 
op nions do not therefore represent official National Institute of Education position or 



Tuckman - 1 

BEST COPY Ammu 

Evaluating Career L'ducationj A Review and Modol 

Bruce W. Tuckman & Joseph A. Carducci 
Rutqers University 

The purpose of this paper is to (1) examine and review evaluations of 
career education programs currently in the literature, and (2) to suggest 
a model or paradigm for conducting such evaluations. The first part of the 
paper will present past work and the second part suggest guidelines for 
future work. 

I. A Review of Evaluative Studies of Career Education 
This review is organized into three sections. In the first, studies are 
described that employed a control or compaHson group with which to compare 
the results of the group receiving the career education experience. We 
call this an experimental or quasi -experimental design. (This section is 
further subdivided to distinguish between those that use both pretest and 
Hottest and those that use posttest only.) The second section of studies 
includes ones employing only a treatment group but using quantitative data, 
typically on a pretest-posttest basis, to evaluate the effects of the treat- 
ment. We call this a descriptive approach. The third section is like the 
second in that only a single group is employed but, unlike the second, only 
qualitative data are obtained. We call these case studies . 
Experiment al and Quasi -Experimental Evaluative Studies 
Pretest-Posttest Desinns^. The most extensive and complete evaluations 
include both an experimental or treatment group - one which receives the 
career education experience, and a control or comparison group which does 
not receive the experience. On the one or more outcome measures employed, 
data are collected at the start of the experience (pretest) and its con- 
clusion (posttera:). These designs, as wc shall see later on. control for 
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various forms of OKperimental bias. Eight ovaluations were found that 
ust?d this approach. 

Cochran and Weis (1972) evaluated the effect of career exploration 
programs on ninth and tenth graders in Dayton, Ohio» at an area vocational 
high school and a general hiqh school. Students from the two program 
schools and a control high school were pre- and posttested on the Ohio 
Matjiinal Interest Survey , the Tennessee Self Concept Scale , and a staff- 
constructed Career Plans Survey . Using analysis of covariance, the results 
showed that students from the vocational school were better informed about 
career choice and demonstrated more positive attitudes toward their abilities 
and chances for vocational advancement as compared to students from the 
control school and general high school. However, changes in vocational 
interests were more closely related to grade level than to school programs. 

Holstein (1972; see also Holstein, 1971 and Olson, 1972) describes 
the evaluation of an exemplary rural area career education program for 
Lincoln County, West Virginia. The project conducted a product evaluation 
for grades 1-6 utilizing an experimental design in which 80 students per 
grade level were randomly assigned to the experimental group and 80 to the 
control group. All students in the study were pre- and posttested in (1) 
language and (2) mathematics achievement using the California Achievement 
Tests and on a homemade test of (3) occupational awareness. Results, using 
an analysis of covariance with pretest scores as covarlates, showed that 
career education students outperformed control students on all measures. 

Hugueley (1973) used the Gordon Occupational Checklist . Crites' 
■Q-Q-reey Maturity Inventor y, and Westbrook's Vocational Maturity Scale to 
compare changes from pre- to postt'ist among students experiencing a career 
education program in the city of Memphis to those not. Results of t-tests 
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showed consistently greater yains for oxperitnentals over controls on the 
two maturity instruments with results on the Checklist being mixed across 
grade levels. 

Ovard (1973) studied to see if there wore differences in attitudes 
toward occupations and the work ethic between students enrolled in a career 
education program (Utah World of Work) and those enrolled in a conventional 
program. Sixth grade students wer pre- and posttested, and thei- scores 
were analyzed by chi square to assess difference* at the beginning and 
end of the school yesr. A teacher questionnaire was also administered at 
the end of the school year. Findings showed teachers were strongly in ' 
support of career education and students were found to show a favorable 
change in attitude toward occupations and the work ethic. 

A Cleveland, Ohio career development program reported by Sims (1973) 
used occupational information, orientation, and career exploration to try 
to bridge the gap between school and earning a living for elementary and 
high school students in an urban poverty area. The product evaluation 
consisted of pre- and posttesting of fifth- and sixth-grade treatment and 
matched control students on a job-informatior. questionnaire with a .85 
reliability. Multifactor analysis of variance showed that the treatment 
schools' students had acquired more job information than the control. A 
process evaluation consisting of a review of materials produced, irite^viaws 
of staff, teachers, and administrators and on-site observations yielded 
the judgment that the program was otherwise not successful in impa^-ting 
more positive attitudes towards work-related concepts or in expandin9 job 
preferences. 

A three-year developmental career education program in Cobb County, 
Georqia is reported on by Smith (1973). The approach fused aspects of 
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vocational oducation, quidance. and a broad-based curnculuin into a systematic 
and sequential effort within the elementary, middle and secondary schools. 
Students in the participating schools and in the control schools were 
tested at the end of one. two. and three years of involvement in the project. 
Tt..^ product evaluation utilized the P^M^ Knowledge Tes.t of Occupations , 
homemade attitude scales and questionnaires, school records, and standardized 
achievement tests. The results (using analysis of variance and chi-square) 
were as follows: (1) no significant difference between treatment and 
control in knowledge of occupations, attitude toward self and others, attitude 
toward the curriculum, additional career counseling, number of dropouts 
with job entry skills, and reduction of dropouts; (2) trends toward lower 
absentee rate at the senior high school and higher academic achievement 
for some of the classes in third, fourth, and sixth grades among treatment 
groups; (3) significant increases for treatment groups in attit'ide toward 
occupational education by teachers, number of students applying to post- 
secondary schools, number of students enrolling in vocational programs ir. 
high school, number of students enrolling in work study and pare.ital 
involvement. Process evaluation using program ratings was highest at the 
elementary and middle school levels and lower at the secondary levels. 

McNulty (1974) studied a Career Opportunities and Practical 
Experiences (COPE) Program involving juniors and senio'.'S at the Ac*-on- 
Boxborough High School in Massachusetts. The study included 100 students 
randomly selected from a larger group of volunteers. The first 50 students 
were assigned to the experimental group and the last 50 to the control 
proup. The COPE program provided for- vocationaT reality testing in the 
community with multiple role models and multiple work ex periences for 
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students at jn appropriate ac|g for exploration. The process evaluation 
conducted via an evaluative questionnaire responded to by participating 
students, parents, sponsors, and teachers revealed a predicted favorable 
view by the participants. Both groups were posttested on the California 
Test, of Personality and Career Maturity Inventory , and an analysis of 
covariance was applied to the data. Analysis indicated that, relative to 
the control qroup, there were no significant gains in personal adjustment, 
social adjustment, and vocational development by the treatment group. A 
positive and significant correlation between scores on the CTP and CMI for 
aW students (N=100) was obtained. 

Warren (1974) conducted a study to determine the change In career 
maturity and job recognition of sixth-, seventh-, and eighth-grade students 
whose teachers participated In career education inservice classes. The 
study was completed in four north central Kansas unified school districts. 
The treatment group involved 407 students and the control gmup 166 students 
all of whom were pre- and posttested on the Attitude Scale and Competence 
Test of Crites' Career Maturity Inventory . T- tests revealed significant 
differences in mean scores on both parts of vhe CMI in favor of the treat- 
ment group. 

The eight studies reviewed above used essentially the same design but 
a variety of outcome measures varying from academic achievement to career 
information to career maturity. Career education was found to produce 
significant effects in contrast to a control consistently but not uniformly 
across studies and across measures. 

P.osttest-Only Desions. Nine studies reported here compared treatment 
and control groups but did not utilize a pretest. This procedurp is 
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vulnerable to certain kinds of bias as described in the second half of the 
paper. 

Boriolo (1972) evaluated the effects of a comprehensive career develop- 
ment program in Hayward, Alameda County, California. The Crites Career 
MMlltjl Inventory was administered to treatment students at the end of the 
program and their results were compared by t-test with students' CMI scores 
from other California schools. No significant differences, (at the .01 
level) were found. However, an Occupational Information Survey^ ore- and 
posttest of treatment and control groups yielded greater gains for treatment 
students than for controls at the elementary level and lesser gains for 
treatment students than for controls at the secondary level.* 

An exemplary career development curriculum project involving 2,424 
elementary and secondary students, 108 teachers, and 180 parents in Knox 
County Schools. Knoxville, Tennessee is summarized in detail by Neal (1972). 
Treatment and control student posttest results revealed no significant 
difference between the two groups on a Knowledge of Careers f^^t. Nor was 
there any correlation between work experience of teachers and the students. 
Randomly chosen parent attitudes wre measured by a questionnaire, and both 
treatment and control parent groups had a favorable attitude toward occupa- 
tional education. A self-concept inventory for grades 1-3 yielded a higher 
mean score for the control groupi grades 4-6 showed mixed results; and in 
grades 7-9 the control group means were higher. Sociograms showed some 
positive changes in treatment students working together. 

Winegard (1972) described a comprehensive vocational guidance program 
in the Indianapolis, Indiana public schools as a possible model for other 
model cities. Using a (1) student opinion poll. (2) Remmer's Purdue Master 

^TT latter result was based on a design utilizing both pretest and posttest 
which fits the preceding subsection. 
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Mi tude Scale . and (3) the LOG Scale, the Occupational Preparation Program 
posttest only results comparing control and treatment students showed 
treatment students to have significantly higher morale, more positive 
attitudes, and a higher level of motivation than controls. 

McCaleb' (1972; see also 1973) describes Project VIGOR in the Portland. 
Oregon schools which addressed itself to the objective of changing a 
conventional academically oriented general education system into one whose 
curriculum reflected the needs of all students from the primary through the 
secondary grades, including post-high-school contact and placement 
assistance, where possible. Evaluation consisted of questionnaires completed 
by random samples of treatment school students and control school students 
at the end of the project. Analysis by t-tests and z-tests indicated that 
the objectives of the project were not met; in many instances, the control 
group appeared to do better than the treatment group on the limited 
instruments. 

Cunningham (1973) assessed the effect of a community-career education 
orientation program on the self-concept of fifth grade pupils in New 
Britain, Connecticut. Extensive resources were committed to a career 
education program in the elementary school by utilizing the community for 
field trips and teacher inservice to create a community-school environment 
for l.'^arning. The self-report How I See Myself Scale was used to measure 
five self-concept variables: academic adequacy, autonomy, interpersonal 
adequacy, physical appearance, and a teacher-school factor among members 
of an experimental and nonequivalent control group. Analysis of test scores 
by t-test to determine the effect on self-concept showed significant 
differences between the control and experimental groups. The null hypothesis 
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that the Coniinunity-Cdrser Education Orientation Program had no significant 
effect on pupil self-concept was not supported. 

Dennard (1973) measured the effects that one year of Career Education 
had on selected fourth grade students in Clayton County, Georgia. The 
treatment group consisted of 84 students and the control group consisted of 
86 fourth graue students. Data were collected on the Iowa Tests of Basic 
Skins., CocjnitivG Abiliti es Test . Student Knowledge of Careers . Personal 
Sketch of Student Career Choice , and Teacher Rating of Career Education . 
Three null hypotheses were rejected: (1) Treatment students with an I.Q. 
of no or below had higher achievement in knowledge of careers at the end 
of grade 4 than comparable control students;. (2) boys had a greater know- 
ledge of careers than girls after one year of .the program; (3) fourth 
graders in the program had a greater knowledge of careers than those not in 
the program. There were no significant, differences in student attitude 
towards careers and teacher ratings of career education in the control and 
treatment schools. 

Holden (1973) reports on an exemplary project in Career Education in 
Fairfield, Chesterfield, and Kershaw Counties of South Carolina. The 
project contained five components: elementary career orientation, worl^ 
experience, interdisciplinary program, intensive training, and job placement. 
A battery of cognitive and affective instruments as well as school records 
was utilized on a posttest-only basis to measure a variety of student 
attitudes, knowledge, and participation among" career education and control 
students. Results showed the following: (1) Kershaw and Chesterfield 
Counties were fairly successful in their efforts to teach elementary school 
students about careers; (2) work experience students had not acquired a 

ERIC 
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sianificantly greater amount of knowledge about occupations; (3) Vocational 
Interdisciplinary Project students were, in several instances, making 
greater achievement in mathematics and language arts than would have 
happened if they had not been enrolled in the VIP Component; (4) the 
attitudes of the VIP and control students toward acquiring a postsecondary 
education, obtaining vocational skills, and curricular offerings were 
generally positive, but not distinguishable from each other; (5) during 
the second program year, VIP students in Chesterfield County scored 
significantly higher on the Metropolitan Achievement Tests than did the 
control group students, whereas during the third year program, the two 
groups' scores were nearly indistinguishable from each other. 

A local Career Education program was evaluated to measure the effect 
of the program on academic achievement and career development by Solomon 
(1973). Forty fifth and sixth grade experimental group students were 
compared by 40 fifth and sixth grade matched controls from another school. 
The students were posttested on the Mathematics and Reading tests of the 
Metropolitan Achievement Tests and the Career Maturity Inventory . No 
significant differences were obtained on any measure. 

The Career Development Exemplary Project (CDEP) examined by Peck 
(1973) was the District of Columbia's effort to place carper development for 
grades K-9 into the mainstream of their curriculum. A career education 
assessment battery was administered to the CDEP students and a control group 
at the conclusion of the project. Results on a recently standardized Self 
Observntion Scale (IDEA) showed CDEP children tc have a better self-concept 
than controls. Results on a project-developed Work Attitud e Survey revealed 
that CDEP students had more positive attitudes toward work thaa controls. 

er|c 
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On the S^h^oJ^ inMHiini^^^^ career development students rev^uil- 

cd more favorable attitudes on most subscales than did controls. The 
■^Iccupjtional VAlues Invpntor^ results for grades 7, 8, and 9 showed that 
project stuGonts were realistic in their career planning. Results on the 
Career Awa^^^ Development Inventory , developed specifically for this 
project, indicated that project students (grades 6-9) were able to relate 
school-learned skills to work situations at a reasonable level of pro- 
ficiency. An Occupational Preference Survey , designed at Washington State 
University, indicated that CDEP students had a greater preference for 
working in association with others than did control students. In spite 
of the number of successful outcomes, process evaluation revealed sporadic 
use and im^llementation of career education materials, limited community 
involvement, and limited quality of project-developed curriculum guides. 

This group of studies, like the preceding ones, shows career education 
to be reasonably effective as reflected on a series of measures of knowledge 
and attitudes. It must be kept in mind, though, that because of the absence 
of a pretest there is no way to be sure that treatment and control students 
began their relative experiences at equivalent levels on the outcome 
measures. This must be considered a weakness of this approach. 

j^crl ptlve Evaluat ions 

In this section, nine evaluations are briefly reviewed each of which 
utilized only the treatment group but applied a variety of outcome measures 
on a pretest-posttest basis. While this approach provides an indication of 
the initial status of students on each measure, It .does not provide a 
basis for comparison of results such as are provided by a control group. 
Rochow (1971) reported on the Vocational Career DGVclopinont Program in 
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Pontiac, Michigan. The- Occupat ional Knowledge Test was administered on a 
pretest and posttest basis to students in kindergarten through grade 6 
In project schools. Using t-tests to analyze the data, student posttest 
means were found to be significantly higher than pretest means. The 
measurement of students' self-concept with the Fantasy . Ability . Reality 
Jest also resulted in significant gain scores. The Crites Career Maturity 
Inventory, given to sixth graders indicated student gain. An extensive 
process evaluation was undertaken and is Included in the report. 

An occupational preparation program presented by Young (1971) 1n 
New Orleans. Louisiana involved all levels of community education. Elementary 
school emphasis was on field trips with experiences to develop realistic 
attitudes toward work; middle school emphasis was on curriculum dealing with 
occupational information and guidance; instruction at the senior high school 
level combined on-the-job training with vocational guidance and job place- 
ment. A pre- and post Attitude Towards Work Inventory was administered to 
project participants only. A significant positive gain (t-test) was found 
for students in grades 1-8. 

An Interdisciplinary career assessment program described by Lapinsky 
(1972) was an experiment in curriculum revision begun in 1969 at Stamford 
High School. Stamford Connecticut, for the purpose of providing Instruction 
relevant to the needs of 86 slow learners and disadvantaged youth. The 
cluster concept and correlation of classroow ir.uterials to occupational 
assessment and skill training were essential to the program. For evaluation, 
a pre- and postexamination was conducted on data gathered from anecdotal and 
cumulative records, questionnaires, inventories, teacher rating scales, the 
Moone^ Problem Checklist , and the SRA Basic Skills in Aritnmetic Trst. Much 
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of the pretesting was conducted after the program was initiated. According 
to the findings, the program met with above-average success. 

Bishop (1973) examined a research and development project in career 
education for Mobile County and Phenix City Public Schools. Alabama. The 
results were tabulated from some pre- and posttest scores of a small random 
sample of program students. The instruments used were the Career Maturity 
Inventory, a self concept scale, and a semantic differential scale. The 
limited testing revealed no significant findings. There was an indication 
that student and teacher attitude toward career education was favorable. 

Gibson (1973) directed a three-year continuous occupational information 
project for grades 1-12 in Henderson County Schools, Kentucky. Results 
on a staff-developed self-concept inventory, the Comprehensive Tests of 
Basjjc Skills , Ohio Vocational Inventory Survey . General Aptitude Test 
BattilD:.. and Barclay Classroom Climate Inventory led the evaluators to 
conclude that the school curriculum had changed from a traditional subject- 
matter orientation to a more life-centered one. 

Hammond (1973) conducted an evaluation of a planned career education 
program for rural students in grades 7-10 in Harrington, Maine which 
provided a variety of experiences and knowledge to form a basis for future 
educational and occupational choices. Paired t.-tests were performed on pre- 
and posttest results of project school seventh and tenth graders taking the 
Occupational Knowledge Test . Results showed significant gain in mean scores 
of seventh grade students but not tenth grade students. Significant 
correlations were obtained between scores on the Lorge- Thorndikc Intelligence 
Test and the Occupa tional Knowled ge Test for both seventh and tenth graders. 
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A career education project was evaluated by Jorgensen (1973) in 
Petersburg City and Radford City. Virginia which aimed at the development 
and implementation of elementary level career awareness and an improved 
quidance, counseling and placement service. Results on a pre- and post- 
homemade career awareness test by K-8 project students indicated no sig- 
nificant increase in occupational knowledge by participating students. 

Lewis (1973) evaluated a career education project in Batesville- 
Leesville, S.C. and used PERT to conclude that the required program 
activities had been carried out. However, pre- and posttesting of occupa- 
tional orientation produced inconclusive results. 

The purpose of a study by Lowe (1973) was to examine selective effects 
of a career education program on upper elementary students in Calhoun 
County, Michigan. The Career Development Battery and the Occupational 
Aspiration Scale were administered to a representative sample of 200 fourth-, 
fifth- and sixth-grade students In the project. Results of analysis of 
variance prompted the following conclusions: In the sample studied, no 
sexual differences existed on knowledge of occupations, awareness of occupa- 
tions, and future occupational aspirations; teacher implementation of the 
career education concept caused differences in student gains on occupational 
knowledge and occupational awareness; and no effects on occupational awareness 
of students were due to parental occupational prestige level. 

Like studies using the more complete designs, these nine studies employ- 
ed a variety of outcome measures and documented, in most cases, significant 
gains by students enrolled in career education programs. A variety of 
attitude and knowledge measures were employed on a pretest and posttest basis, 
but significant gains are difficult to evaluate in the absence ot comparison 
data. 
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Ca se Studies 

Many career education programs meet the requirement for evaluation 
by administering a questionnaire to students to solicit their opinions and 
feelings toward the program and possible recommendations for improvement 
(Krueck, 1972; Kunzman, 1973). Other evaluations are broader in scope in 
terms of those whose opinions and reactions are solicited. Questionnaires 
are distributed not only to students but to teachers, parents, and 
community members in an effort to estimate support; the overall impression 
of the evaluator reading the responses is reported (Jacobs, 1971; Mietus 
& Stilling. 1971; Nelson, Gage, & Walter, 1971; Carlson. 1972, 1973; 
McKinney, 1973). Some evaluations focus on a description of the process of 
the program - for example, the degree of utilization of program materials 
by teachers {Moskowitz, 1973) - rather than on the outcomes. Other 
evaluations use little if any instrumentation (Berg, 1972), and still others 
use extensive instrumentation but only summarize the results qualitatively 
(Willis, 1973 a, b) rather than providing any statistical analyses. 

The prototype of the case study evaluation relies on the use of an 
outside evaluator or team of evaluators. This person (or persons) will 
usually spend from one to five or more days observing the program and talking 
to students, teachers, and possibly parents and community members. Informa- 
tion will also be gleaned from written documents and conversations with the 
program director. The impressions of the team members are then written up 
in the form of a narrative which may or may not be organized around a given 
set of questions or a prescribed outline (Jenkins, 1971; Educational 
Manaqoment Services, 1973; KlauS, T973; Nerden, 1973; Riley, 1973; 
Schuennann, 1973; Shelford, 1973). Within the narrative, the evaluators 
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report their conclusions concerning the effectiveness of the program 
usually in terms of their judgment of whether the project has met its 
objectives. 

Invariably case studies find the programs evaluated to be successful, 
and the above studies*- clearly more favorable than unfavorable - were 
no exception. Whether. this accurately reflects the strengths of career 
education or the vulnerability of the case study approach to subjectivity 
and bias cannot be determined, but. at any rate, the results of case studies 
such as these must be interpreted with caution. 
II. A Framework for Assessing Career Education Utilization Programs 
For the assessment of the adequacy or completeness of career education 
programs (or any other educational programs, for that matter) a framework 
or category system is helpful. A framework Is also useful as a basis for 
making recommendations for subsequent evaluation research. Sucl. a frame- 
work appears in Table 1. 
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Table 1 

Framework for Assessing Career Education Utilization 
Programs (FACE UP) 

I. Design 

a. experimental and quasi -experimental : experimental vs. 
control group 

b. descriptive: one group - quantitative data 

c. case study: one group - qualitative data 

II. Independent Variable - nature of treatment 

a. intensity d. public relations 

b. duration e. quality of materials 

c. teacher t»^aining f. grade level 

III. Dependent Variables - outcomes 

A. Process 

a. teaching behavior c. counseling 

b. student behavior 

B. Product 

a. self-awareness (self-descriptions, self-concept) 

b. career-awareness (knowledge of careers) 

c. decision-making skills (ability to make decisions and 
solve problems) 

d. overall school behavior (grades, school achievement, 
attendance, attitudes toward school) 

e. career competencies (salable skills) 

f. career attitudes and interests (attitudes toward work, 
career development, interests) 

C. Follow-up 

IV. Control Variables 

a. selection bias 

b. history bias 

c. maturation bias 

d. population restrictions 

V. Data Analysis 

a. descriptive 

b. inferential 
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D esign 

The first question th.> "s asked about an evaluative study deals with 
the design. In an experimental or quasi -experimental study, a group or 
sample of students experiencing the program is compared with a group or 
sample not experiencing the program or experiencing an alternative program. 
When assignment to experimental and control groups is on a random basis, 
the design is experimental; when intact or preassigned groups are compared, 
the design is quasi-experimental. The advantage of these designs is that 
it makes possible inferences about cause and effect. That is, you can 
separate the effect of the program from the effect of other nonprogram 
variables since the control group experiences these other variables when 
it is not experiencing the program. These designs are recommended because 
of this important feature. Without comparison data, who is to say whether 
the same results might not have been obtained without the program? 

Descriptive and case-study approaches lack a control group and are 
themselves distinguished by the presence or absence of quantitative data. 
The advantage of quantitative data is that it lessens the subjectivity 
with which outcomes are described. Both designs, however, are limited in 
their conclusiveness because of the absence of comparison data. Where data 
from previous school years can be used for comparison purposes, the desiyn 
can be more appropriately categorized as quasi-experimental. 

,E valuators are urged to. use. experimental and quasi -experimental designs 
with the inclusion of pretest^ data . Where a descriptive approach cannot 
be avoided, pretests are a must if any conclusions about gains, and hence 
program effectiveness, are to be drawn. Case studies by themselves cannot 
be considered reliable or valid sources of evaluative data. A major 
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deficiency of many of the studies reviewed in the first part of this paper 
is their reliance on the weaker designs. 

The Treatment 

The second consideration in examining the evaluative studies is the 
treatment to be evaluated. The term career education , for example, covers 
a lot of ground. It covers programs that are experienced daily and others 
only weekly; some that are experienced in class and others after school. 
Its programs may range from field trips to skill training. It can operate 
in elementary school or in high school. The reader must know what is being 
evaluated. Naturally, the less the intensity, duration, or quality of the 
treatment the less likely it is to have an effect. Evaluators are encouraged 
to describe treatments fully but economically ; 1t is not the number of 
words of description but the amount of important information they convey. 
The use of, a set of descriptors like those offered in Table I (II, a-f) may 
be helpful for this purpose. The studies reviewed devoted considerable 
space to the description of treatment (although we afforded little space 
to it in our summaries). There is no problem with the amount of description; 
however, the use of descriptors might have helped to communicate it more 
eas i ly . 

Outcomes 

The outcome area is a major concern in this paper since measurement 
is its primary focus. Outcomes can be conveniently subdivided into process — 
that Is, teacher, student, and counselor behavior in the conduct of the 
program, and product-student changes that result from the program. 

Process . Process outcomes, perhaps the less important of the two 
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subcatepories, deal primarily with the behavior of teachers in carrying out 
the program, often as reflected in the behavior of students. We might, for 
example, be concerned about whether teachers encouraged hands-on experience, 
question-asking, integrating of subject-matter and experience, and so on. 
While this might be considered either part of the treatment or part of. the 
product, it seems to fall between the two - that is, neither built into 
the program nor necessarily part of its learner objectives. 

It ma^ be hel pful for evaluators to measure process outcomes primari ly 
as a way of bridging the gap between input and output. Processes that are 
inconsistent with program philosophy and intent are not likely to yield 
the products called for in the objectives. To measure process, the evaluator 
will probably require homemade, observational instruments geared to the 
intentions of the program or to good pedagogical practice. In come areas of 
educational innovation, such as the open classroom, instruments for measuring 
process have been developed (for example, see Tuckman, Cochran, & Travers, 
1974). Because of the variability in career education programs, :;uch 
instrument development may be unlikely. Nevertheless, systemati c observa- 
tioji of classroom behavior is, an important component of the evaluation of 
career education and should not be overlooked. Unfortunately, systematic 
process evaluation is absent in much of the work reported in the first 
part of this paper. 

P^o<i^ct. A most critical area in any evaluation under- taking is 
the determination of products to be measured and instrumentation to be 
employed for their measurement. Two weaknesses often encountered in this 
aspect of evaluation are (a) a too limited view of product outcomes and (b) 
a high variability from study to study in the product measures employed. 
The pr(?viously reviewed studies suffer from both deficiencies. 
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Table 1 includes a listing of six product areas with some suggested 
variables for measurement in each. It would be possible to identify 
additional areas but the six that seem of paramount importance as reflections 
of the adequacy of career education are: 1) self-awareness. 2) career- 
awareness, 3) decision-making skills, 4) overall school behavior, f) 
career competencies, and 6) career attitudes and interests. It. is recommend- 
ed that, evaluations, of career education include measures in, as, man^ of the 
six, areas, as possible. In addition, if projects include objectives out- 
side of these areai. measures to cover these objectives will also be required. 

The self-awareness area includes what students know about themselves 
and their attitudes toward themselves. The latter, usually referred to as 
self-esteem or self-concept» can be measured by .a variety of existing 
instruments such as the Tennessee Self Concept Scale . Piers- Harris Children's 
Self Concept Scale, or Self- Appraisal Inventory .* Measures depicting the 
degree of self-awareness must typically be designed by evaluators themselves 
and are usually not included in studies. 

Self-awareness might also be studied by having the student describe 
his or her own talents and characteristics and then, by comparing that 
information to the same information obtained from tests, arriving ai a 
measure of accuracy. An interview procedure might also be used effectively 
to determine how well students know themselves. Finally, self-awareness 
might be taken to mean behaviors aimed at finding out about oneself and 
could be determined by means of a self-report of behavior wherein students 

niiTTirst two are published by Counselor Recording and Tests, Box •6184 - 
Acklen Station, Mc-^shville, Tenn. 37212. The third is distributed by th« 
Instructional Objectives Exchange, Box 24095, Los Angeles, Calif. 90024 
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indicate the kinds of behavior they have engaged in to uncover their own 
talents and characteristics. Many of the studies reviewed included 
measures in this area, often homemade. 

The measurement of career-awareness, meaning knowledge of career 
characteristics and requirements, necessitates a cognitive instrument deal- 
•ing with specific careers. In terms of general career-awareness, the 
problem becomes one of deciding which careers to ask questions about; in 
terns of specific career-awareness, the difficulty lies in finding or 
building tests in the large number of career areas that students might choose 
to pursue. Instrumentation in this area is also hard to find. Nevertheless, 
a large number of the reviewed studies included such a measure. 

Decision-making skills is probably the most difficult and most overlooked 
of the recommended product areas. Few if any evaluations (including the 
ones reviewed) contain measures of it although it is avowedly an important 
objective of career education. Among existing tests, perhaps reasoning 
tests came closest, but they are more measures of intelligence than of learn- 
ed skills. For older students, the Watson_-_61a2er Critical Thinking 
Appraisal* could be used in this area. Certainly the area of decision- 
making skills is one that calls for test development. 

Of the six categories, measures of overall school behavior are the 
easiest to find and are measured in many studies (including many of those 
reviewed) as a by-product of career education. Most schools routinely 
administer standardized achievement test batteries, the results of which can 
be incorporated into a career education evaluation. If the same battery 
has been used in previous years, pretest data are also available. It is 
probably easiest when test administration occurs in the spring since 

*Th1s test is" publislied by Harcourt Brace Jovanovich. 
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evaluators usually complete data collection at the end of the school year. 
Information about grades, attendance, dropping out and disciplinary actions 
are typically part of the normal record-keeping process of schools. If 
behavior ratings of students are also routinely made, these may also be 
included in the evaluation. Finally, measures of attitudes toward school 
are available (see, for instance, the School Sentiment Index*) and can be 
easily constructed for evaluative purposes. 

Many career education programs in the advanced grades aim to provide 
students with salable skills upon graduation. While trade competency 
measurement is available in some states, it is usually not for entry level 
skills. The best way to assess high school exit skill levels may be to 
use ratings of student performance by shop teachers on suitable rating forms 
constructed for this purpose. What this measure might lack is reliaoillty 
it would make up for in both appropriateness (i.e., content validity) 
and usability (i.e., practicality). Little evidence of such measurement 
appeared in the studies reviewed. 

Finally, measures of various career attitudes and interests are 
available for use and can be located through test compendia such as the 
Mental measurement yearbook (Buros, 1972). Instruments such as the Career 
^.^^"''Tty Inventory and OMo Vocational Interest Survey* * may be used for 
measurement of career attitudes and interest and. in fact, were in common 
evidence in many of the evaluation studies reviewed. Interest measures muy 
be dimensional ized by examining discreteness of interest patterns or degree 
of interests in areas covered in the program. (Awareness of interest would 

'ims measure is distributed by the Instructional Objectives Exchange. 

**ThG CMI is published hy California Test Bureau/McGraw H^ll; the OVIS 
by Harcourt Brace Jovanovich. 
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be covered under self-awareness. ) Descriptions of measures such as those 
mentioned above as well as lists of instrument reference sources may be 
found in Tuckman (1975). 

''Q^^Q'^""P ' ultimate evaluation of career education programs will 
be through follow-up of graduates. Longitudinally detennined Outcomes such 
as employment status and career adjustment and satisfaction, measured at 
various postprogram intervals, will tell us much about the kinds of effects 
that we can expect from career education programs. Perhaps the most 
economical and. hence, likely way of obtaining follow-up data is to include 
a question about career education experiences in all general large-scale 
follow-up undertakings such as Project TALENT. While such data will be 
retrospective, they will help us determine the long-range effects of career 
education. Follow-up data were all but absent 1n the reviewed studies. 

Control Variables 

It is important to be able to separate the effects of career education 
from the effects of both selection and history bias in our evaluations. 
Selection bias refers to outcomes that are more a function of student 
characteristics such as mental ability than of career education experiences 
£er se. Tuckman and Montare (1974). for example, found that students in the 
so-called college preparatory program possessed more career information than 
students in the vocational programs in two school districts. More tuan 
likely, this is a reflection of the higher mental ability of the former group 
than of any specific in-school experiences. To control for these differences, 
adjustments via either design or -statistics must be made. Where experimental 
and control. £roups are to be compared, care must be t.akpn if^ on,.;,-:,",^ fu«n. 
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on G2iMiilL)i innuential i ndividual differences . To acconip 1 1 sh th i s . 
pretesting js. iianjjilted' The studios reviewed that use experimental or 
quasi -experimental designs on a posttest-only basis make it difficult to 
equalize groups or to assess their differences at the start of the program. 

Individual differences can also be somewhat controlled by restricting 
the population to be studied. Limiting comparisons, for example, to students 
who are employment-bound allows the range of Individual differences to 
be narrowed and their Influence to be lessened. Of course, evaluators doing 
this must recognize that their ability to generalize will also be limited 
to the population studied. However, if individual differences will other- 
wise be uncontrolled, the sacrifice in the ability to generalize is a price 
well worth paying for increased internal validity. 

History bias refers to the influence of other events occurring at the 
same time as the experience or treatment being evaluated. Students have 
a wide variety of educational experiences simultaneously making it difficult 
to isolate the effects of specific career education experiences from othe- 
educational effects. Maturation bias is also a potential threat to 
validity in these evaluations since students are maturing during the treat- 
ment period, such maturation producing gains of its own. Both history 
and maturation bijs, can be effectively controlled evaluatov-s utilize a 
control or comparison group who receive all the same eoucational expe n'ences 
as. the treatment group , except for the treatment and w!]0 are the same pje 
as treatment itudents.. Descriptive studies such as those reviewed fall 
short on this requirement. 

Statistics 

Where quantitative data are obtained, both descriptive and inferential 
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statistics should be used. Statistics help the reader determine the extent 
to which findings are true or spurious. This is true whether one is 
assessing gains from pretest to posttest using simple t-test results or 
carrying out more sophisticated analyses or covariance, comparing groups 
while statistically controlling for individual differences, jn. all cases , 
evaluators are urged to analyze their data statistically . 



The use of the above guidelines and recommendations should reduce the 
error in career education evaluations thereby allowing the results to be 
a more clearcut reflection of the efficiency of the programs themselves. 
The reader is also referred to Tuckman (1972) for more detailed information 
about evaluation. 
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